Subject priority based image capture

ABSTRACT

An image capture device obtains first image data from a front facing camera on a front face of the device including a display. The first image data represents a subject within a field of view of the image capture device. Facial recognition is performed in response to the first image data from the front facing camera. The front image data is used for determining whether any subject is a priority subject, of whom at least one priority subject image-indicative data is accessible by the image capture device. A region of interest is selected, corresponding to the subject determined to be a priority subject. Automatic focus, automatic exposure, or automatic white balance is performed using the selected region of interest. Second image data from the front facing camera are captured based on the automatic focus, automatic exposure, or automatic white balance using the selected region of interest.

RELATED APPLICATIONS

This application is a continuation of, and claims priority to,application Ser. No. 15/969,237, filed May 2, 2018, the disclosure ofwhich is incorporated herein by reference in its entirety.

BACKGROUND Field of the Disclosure

This disclosure relates generally to imaging devices and, morespecifically, to automated image capture control.

Description of Related Art

Digital image capture devices, such as cameras in cell phones and smartdevices, use various signal processing techniques in an attempt torender high quality images. For example, these image capture devicesautomatically focus their lens for image sharpness, automatically setthe exposure time based on light levels, and automatically adjust thewhite balance to accommodate for the color temperature of a lightsource. In some examples, image capture devices include facial detectiontechnology. Facial detection technology allows the image capture deviceto identify faces in a field of view of an image capture device's lens.The image capture device may then apply the various signal processingtechniques based on the facial identifications.

SUMMARY

According to one aspect, a method for controlling an image capturedevice comprises obtaining first image data from a camera of the imagecapture device. The first image data represents one or more subjectswithin a field of view of the image capture device. The method includesperforming facial recognition on the first image data. Further, andbased on the facial recognition, the method includes determining thatthe one or more subjects include a particular subject based on prioritysubject image-indicative data. The method also includes selecting aregion of interest corresponding to at least the particular subject.Further, the method includes processing a second image based on theselected region of interest.

According to another aspect, an image capture device comprises a memorycontaining priority subject image-indicative data and at least oneprocessor. The processor is coupled to the memory for accessing theimage-indicative data. The processor is configured to obtain first imagedata from a camera of the image capture device, where the first imagedata represents one or more subjects within a field of view of the imagecapture device. The processor is configured to perform facialrecognition on the first image data and determine, based on the facialrecognition, that the one or more subjects include a particular subjectbased on priority subject image-indicative data. The processor isconfigured to select a region of interest corresponding to at least theparticular subject. The processor is configured to process a secondimage based on the selected region of interest.

According to another aspect, a non-transient computer-readable storagemedium comprises computer-executable instructions stored tangiblythereon. The instructions, when executed by one or more processors,cause the one or more processor to: obtain first image data from acamera of the image capture device, where the first image datarepresents one or more subjects within a field of view of the imagecapture device; perform facial recognition on the first image data;determine, based on the facial recognition, that the one or moresubjects include a particular subject based on priority subjectimage-indicative data; select a region of interest corresponding to atleast the particular subject; and process a second image based on theselected region of interest.

According to another aspect, an image capture device comprises: meansfor obtaining first image data from a camera of the image capturedevice, where the first image data represents one or more subjectswithin a field of view of the image capture device; means for performingfacial recognition on the first image data; means for determining, basedon the facial recognition, that the one or more subjects include aparticular subject based on priority subject image-indicative data;means for selecting a region of interest corresponding to at least theparticular subject; and means for processing a second image based on theselected region of interest.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram of an exemplary image capturing device havingpriority subject based automatic focus, automatic exposure, or automaticwhite balance control;

FIG. 2 is a flow chart of a method for selecting a region of interest(ROI) in a field of view (FOV) using the exemplary image capturingdevice of FIG. 1;

FIG. 3A is an image showing subjects in a FOV of the exemplary imagecapturing device of FIG. 1;

FIG. 3B illustrates the image of FIG. 3A with a ROI identified thatincludes the faces of the subjects in the FOV;

FIG. 3C illustrates the image of FIG. 3A with an adjusted ROI identifiedthat includes the face of a priority subject of the subjects in the FOV;

FIG. 4A is an image showing subjects in a FOV of the exemplary imagecapturing device of FIG. 1;

FIG. 4B illustrates the image of FIG. 4A with a ROI identified thatincludes the faces of the subjects in the FOV;

FIG. 4C illustrates the image of FIG. 4A with an adjusted ROI identifiedthat includes faces of a subset of the priority subjects in the FOV;

FIG. 5 is a flowchart of an exemplary method that can be carried out bythe exemplary image capturing device having priority-subject basedregion of interest selection for automatic focus, automatic exposure, orautomatic white balance control of FIG. 1;

FIG. 6 is a flowchart of another exemplary method that can be carriedout by the exemplary image capturing device having priority region ofinterest selection for subject based automatic focus, automaticexposure, or automatic white balance control of FIG. 1; and

FIG. 7 is a flowchart of another exemplary method that can be carriedout by the exemplary image capturing device having region of interestselection for priority subject-based automatic focus, automaticexposure, or automatic white balance control of FIG. 1.

FIG. 8 is a flowchart of a method that activates face recognition upondetection of a fingerprint of a primary user of the image capturedevice.

FIG. 9 is a flow chart showing additional steps that can be added tosome embodiments of the method of FIG. 6.

DETAILED DESCRIPTION

While the present disclosure is susceptible to various modifications andalternative forms, specific embodiments are shown by way of example inthe drawings and will be described in detail herein. The objectives andadvantages of the claimed subject matter will become more apparent fromthe following detailed description of these exemplary embodiments inconnection with the accompanying drawings. It should be understood,however, that the present disclosure is not intended to be limited tothe particular forms disclosed. Rather, the present disclosure coversall modifications, equivalents, and alternatives that fall within thespirit and scope of these exemplary embodiments.

Many cameras are equipped to identify faces in the field of view (FOV)of the camera, and select a lens position that provides the best focusvalue for a region of interest (ROI) containing all of the identifiedfaces. Many times, however, the selected lens position does not resultin an optimal captured image for a particular face or subset of thefaces in an image containing several faces.

This disclosure provides an image capture device having one or more ofpriority subject-based automatic focus (AF), automatic exposure (AE), orautomatic white balance (AWB) control, and corresponding methods. Theimage capture device can identify subjects within its field of view(FOV), and determine whether any one or more of the identified subjectsis a priority subject. A priority subject can be, for example, an owneror other primary user of the image capture device, or another person oranimal designated as a priority subject, in a manner described below. Ifany one of the identified subjects is a priority subject, the imagecapture device can determine a region of interest (ROI) that includesthe priority subjects. For example, the region of interest can includethe priority subjects and exclude all subjects other than prioritysubjects. The image capture device can then adjust one or more of AF,AE, or AWB of the image capture device based on image data within theROI, which includes the priority subject(s). In this description, unlessexpressly stated otherwise, the ROI refers to the region of interestthat is used during AF, AE, and/or AWB.

In this description, the terms “priority subject” and “prioritizedperson” are used interchangeably to refer to person or animal (e.g., apet), identified by a user of the camera as being prioritized, and forwhom the image capture device contains one or more images to supportface recognition. A prioritized person is an example of a prioritysubject, and in the examples described herein, the image capture deviceis equally capable of applying the same methods to animals, or statuesor paintings including faces.

Among other advantages, the image capture device is capable of providingautomated image capture enhancements in consideration of lightingconditions affecting the priority subjects. For example, the imagecapture device can automatically optimize one or more of AF, AE, or AWBbased on image data within its field of view that represents prioritysubjects. For example, a primary user of the image capture device can bea predetermined priority subject. When the primary user of the imagecapture device takes a picture of themselves (e.g., a selfie), the AF,AE, and/or AWB settings are optimized for the ROI containing the primaryuser of the image capture device, regardless of others who may appear inthe picture. Instead of selecting a compromise lens position thatminimizes total focusing error of all the faces in the FOV, the lensposition is selected to minimize the focusing error of the prioritizedface(s). Thus, priority subjects receive optimal camera settings whenthe image capture device takes their picture. For example, when usingthe front camera for a selfie, the image capture device can ensure thatthe owner of the camera is in focus, even if other faces in the imageare out of focus. Similarly, exposure and/or white balance may also beoptimized for the priority subjects instead of averaged or otherwiseweighted across other subjects.

FIG. 1 is a block diagram of an exemplary image capture device 100. Thefunctions of image capture device can be implemented in one or moreprocessors, one or more field-programmable gate arrays (FPGAs), one ormore application-specific integrated circuits (ASICs), one or more statemachines, digital circuitry, any other suitable circuitry, or anysuitable hardware. In this example, image capture device 100 includes atleast one processor 160 that is operatively coupled to (e.g., incommunication with) camera optics and sensor 115 for capturing images.Processor 160 is also operatively coupled to instruction memory 130,working memory 105, input device 170, and storage medium 110. Inputdevice 170 can be, for example, a keyboard, a touchpad, a stylus, atouchscreen, or any other suitable input device. In some examples,processor 160 is also operatively coupled to one or more of GlobalPositioning System (GPS) unit 116, transceiver 117, and display 125.

The image capture device 100 can be implemented in a computer with imagecapture capability, a special-purpose camera, a multi-purpose devicecapable of performing imaging and non-imaging applications, or any othersuitable device. For example, image capture device 100 can be a portablepersonal computing device such as a mobile phone, digital camera, tabletcomputer, laptop computer, personal digital assistant, or any othersuitable device.

Although this description refers to processor 160, in some examplesprocessor 160 can include one or more processors. For example, processor160 can include one or more central processing units (CPUs), one or moregraphics processing units (GPUs), one or more digital signal processors(DSPs), one or more image signal processors (ISPs), one or more deviceprocessors, and/or one or more of any other suitable processors. In someexamples, processor 160 can include different types of processors. Forexample, processor 160 can include an image signal processor 120 and adevice processor 150. In this example, image signal processor 120 canperform various image capture operations on received image data toexecute AF, AE, and/or AWB. Device processor 150 can perform variousmanagement tasks such as controlling display 125 to display capturedimages, or writing to or reading data from working memory 105 or storagemedium 110. Device processor 150 can also configure image captureparameters that can be used by image signal processor 120 to captureimages, such as AF, AE, and/or AWB parameters. Although in FIG. 1processor 160 is located within image capture device 100, in someexamples, processor 160 can include one or more cloud-distributedprocessors. For example, one or more of the functions described belowwith respect to processor 160 can be carried out (e.g., performed) by aremote processor, such as a cloud processor 148 within a cloud-basedserver, where the cloud processor 148 can be connected to the processor160 via a network 151. The cloud processor 148 can be coupled tonon-transitory cloud storage media 149, which may be collocated with, orremote from, the cloud processor 148. The network 151 can be anypersonal area network (PAN), local area network (LAN), wide area network(WAN) or the Internet.

Camera optics and sensor 115 can include one or more image sensors andone or more lenses to capture images. Processor 160 can control cameraoptics and sensor 115 to capture images. For example, processor 160 caninstruct camera optics and sensor 115 to initiate an image capture(e.g., take a picture), and can receive the captured image from cameraoptics and sensor 115. Camera optics and sensor 115, storage 110, andthe processor 160 provide a means for capturing second image data fromthe front facing camera based on the at least one of automatic focus,automatic exposure, or automatic white balance using the selected regionof interest.

Instruction memory 130 can store instructions that can be accessed(e.g., read) and executed by processor 160. For example, instructionmemory 130 can include read-only memory (ROM) such as electricallyerasable programmable read-only memory (EEPROM), flash memory, aremovable disk, CD-ROM, any non-volatile memory, or any other suitablememory.

Processor 160 can store data to, and read data from, working memory 105.For example, processor 160 can store a working set of instructions toworking memory 105, such as instructions loaded from instruction memory130. Processor 160 can also use working memory 105 to store dynamic datacreated during the operation of image capture device 100. Working memory105 can be a random access memory (RAM) such as a static random accessmemory (SRAM) or dynamic random access memory (DRAM), or any othersuitable memory.

In this example, instruction memory 130 stores capture controlinstructions 135, AF instructions 140, AWB instructions 141, AEinstructions 142, image processing instructions 143, subject detectorinstructions 144, priority subject selector instructions 146, ROIadjuster instructions 147, and operating system instructions 145.Instruction memory 130 can also include additional instructions thatconfigure processor 160 to perform various image processing and devicemanagement tasks.

AF instructions 140 can include instructions that, when executed byprocessor 160, cause a lens of camera optics and sensor 115 to adjustits lens position. For example, processor 160 can cause a lens of cameraoptics and sensor 115 to adjust so that light from a region of interestwithin a field of view (FOV) of the imaging sensor is focused in a planeof the sensor. The selected ROI can correspond to one or more focuspoints of the AF system. AF instructions 140 can include instructionsfor executing autofocus functions, such as finding the optimal lensposition for bringing light from a region of interest into focus in theplane of a sensor. Autofocus can include, for example, phase detectionautofocus (PDAF), contrast autofocus, or laser autofocus. PDAF dividesincoming light into pairs of images and captures the divided light rayscoming from the opposite sides of the lens, creating a rangefinder. Thetwo images are then analyzed to find a separation (phase) error anddetermine whether the region of interest of the sensor is in focus inthe sensor's plane. Contrast AF moves a lens through its position range,stopping at a point where maximal contrast is detected between adjacentpixels at an edge in the FOV. Laser AF emits a light from a laser orlight emitting diode (LED) on the subject and calculates a distance tothe subject based on how long it takes the light to reach the subjectand return.

AWB instructions 141 can include instructions that, when executed byprocessor 160, cause processor 160 to determine a color correction to beapplied to an image. For example, the AWB instructions 141, whenexecuted by processor 160, can cause processor 160 to determine anaverage color temperature of the illuminating light source under whichan image was captured, and to scale color components (e.g., R, G, and B)of the image so they conform to the light in which the image is to bedisplayed or printed. In some examples, the AWB instructions 141, whenexecuted by processor 160, can cause processor 160 to determine theilluminating light source in a region of interest of the image. Theprocessor 160 can then apply a color correction to the image based onthe determined color temperature of the illuminating light source in theregion of interest of the image.

AE instructions 142 can include instructions that, when executed byprocessor 160, cause processor 160 to determine the length of time thatsensing elements, such as an imaging sensor of camera optics and sensor115, integrate light before an image is captured. For example, processor160 can meter ambient light, and select an exposure time for a lensbased on the metering of the ambient light. As the ambient light levelincreases, the selected exposure time becomes shorter. As the ambientlight level decreases, the selected exposure time becomes longer. In thecase of a digital single-lens reflex (DSLR) camera, for example, AEinstructions 142, when executed, can determine the exposure speed. Insome examples, processor 160 can meter the ambient light in a region ofinterest of the field of view of a sensor of camera optics and sensor115.

A 3A engine 136 provides a means for performing at least one ofautomatic focus, automatic exposure, or automatic white balance of theimage capture device based on using the selected region of interest. Insome examples of image capture devices 100, the three sets ofinstructions, AF instructions 140, AWB instructions 141 and AEinstructions 142, are included in “3A” engine 136. The 3A engine 136 caninclude instructions that cause processor 160 to operate on raw imagesensor data measured by an image sensor of camera optics and sensor 115prior to capturing an image.

Capture control instructions 135 can include instructions that, whenexecuted by processor 160, configure processor 160 to adjust a lensposition, set an exposure time, set a sensor gain, and/or configure awhite balance filter of the image capture device 100. Capture controlinstructions 135 can further include instructions that, when executed byprocessor 160, control the overall image capture functions of imagecapture device 100. For example, capture control instructions 135, whenexecuted by processor 160, can cause processor 160 to execute AFinstructions 140 to calculate a lens or sensor movement to achieve adesired autofocus position and output a lens control signal to control alens of camera optics and sensor 115.

Image data of priority subjects can be stored in a means for storing atleast one priority subject image-indicative data corresponding to one ormore priority subjects. The means for storing can be a non-transitory,machine-readable storage medium 110 or cloud storage 149. A prioritysubject can include, for example, an owner or other user of the imagecapture device 100, a person, an animal, or an object. Processor 160 candetermine whether detected subjects are predetermined priority subjectsbased on previously stored image data corresponding to predeterminedprioritized persons (e.g., the primary user, the owner, the owner'schild or, or the owner's spouse) or animals (e.g., the owner's pet).Image data corresponding to a predetermined prioritized person can bestored, for example, in storage medium 110. In some examples, a user ofimage capture device 100 can provide image data associated withpredetermined priority subjects to image capture device 100 by capturingone or more images of the subject with the image capture device 100 anddesignating the subject of the images as a priority subject, or bydownloading images of the priority subject from another device ornetwork.

The means for storing can be a local storage medium 110, such as a harddrive, a solid-state memory, or a FLASH memory, for example. The meansfor storing can also be a remote storage medium, such as cloud storage149, which can be embodied in a cloud-based server memory, a memorydevice on another image capture device 100, a networked computer, or anyother suitable remote storage device. Storage medium 110 can includepriority subject data 165 comprising image data for one or morepredetermined prioritized persons. For example, priority subject data165 can include one or more images for each prioritized person. In someexamples, priority subject data 165 includes multiple imagescorresponding to multiple views for each prioritized person, forexample, a plurality of images from a plurality of different angles. Forexample, the multiple images can include a front image of thepredetermined prioritized person, a right side image of thepredetermined prioritized person, and a left side image of thepredetermined prioritized person.

In the examples described below, the priority subject data 165 includesimage data. In other embodiments, the priority subject data 165 caninclude other types of priority subject image-indicative data, which areindicative of image data, where the priority subject image-indicativedata are based on (e.g., extracted from, derived from or representativeof) image data of a priority subject. For example, the priority subjectdata 165 can include a plurality of extracted facial features and/orfiducial-point-based face graph data for the priority subject. In otherembodiments, the priority subject image-indicative data in prioritysubject data 165 can include holistic data for the priority subject,such as an array of intensity values or dimensionality-reduced eigenfacepicture data in a principle coordinate (latent variable) system. Inother embodiments, the priority subject image-indicative data ofpriority subject data 165 can include coefficient data for a neuralnetwork trained to identify members of a predetermined set of prioritysubjects, such as faces. These are only examples of priority subjectimage-indicative data used in exemplary pattern recognition (e.g., facerecognition) techniques, and are not exclusive. For brevity, the term,“image-indicative data” is used below to refer to image data and/orother forms of data based on, extracted from, or representative of imagedata. The image-indicative data include, but are not limited to, data towhich a pattern recognition technique can be applied, as well as RAW,intermediate or partially-processed image sensor data.

For example, if the priority subject image-indicative data compriseextracted facial features and/or fiducial-point-based face graph data,then the pattern recognition (e.g., facial recognition) includesextracting facial features and/or fiducial point based graph data fromthe image data of the subject(s) in the FOV of the image capture devicefor comparison with the priority subject image-indicative data in thepriority subject data.

If the priority subject image-indicative data comprise an array ofintensity values, then the facial recognition includes determining anarray of intensity values based on the image data from the subject(s) inthe FOV of the image capture device for comparison with the prioritysubject image-indicative data in the priority subject data.

If the priority subject image-indicative data comprisedimensionality-reduced eigenface picture data in a principle coordinate(latent variable) system, then the facial recognition includestransforming image data from the subject(s) in the FOV of the imagecapture device to the principal coordinates (latent variables) forcomparison with the priority subject image-indicative data in thepriority subject data.

If the priority subject image-indicative data include coefficient datafor a neural network trained to identify members of a predetermined setof priority subjects' faces then the facial recognition includesinputting the image data from the subject(s) in the FOV of the imagecapture device to the neural network and receiving from the neuralnetwork a value indicating a probability that the subject is one of thepriority subjects.

In the example of FIG. 1, the priority subject data 165 is storedlocally in the image capture device 100. In other embodiments, thepriority subject data 165 is stored remotely. For example, the prioritysubject data can be stored in cloud storage 149 coupled to cloudprocessor 148, and the processor 160 can access the priority subjectdata using a communications protocol, such as hypertext transportprotocol (HTTP). In other embodiments, the priority subject data 165 isinitially stored locally in image capture device 100, and is copied tocloud storage 149. If the user has multiple image capture devices, theuser can access the priority subject data in from cloud storage. In someembodiments, face detection and/or face recognition are initiated byprocessor 160 and performed remotely by cloud processor 148, using thecopy of priority subject data 165 in cloud storage 149.

Subject detector block 144 provides means for initiating facialdetection on first image data representing one or more subjects within afield of view of the image capture device to detect one or more faces.Subject detector instructions 144 can include instructions that, whenexecuted by processor 160, cause processor 160 to detect one or moresubjects in a field of view of a lens of camera optics and sensor 115 asbeing faces. For example, processor 160 can obtain raw image sensor dataof an image in a field of view of a lens of camera optics and sensor115. In some embodiments, processor 160 can initiate face detection anddetermine if one or more subjects are in the field of view by, forexample, performing the facial detection locally within processor 160.In other embodiments, processor 160 can initiate remote performance offace detection by transmitting a request to a cloud processor 148 orother remote server. The request causes the cloud processor 148 or otherremote server to perform the computations to determine if the field ofview of image capture device 100 contains one or more faces, and respondto the processor 160 with identification of region(s) in the FOVcontaining a face.

Priority subject selector block 146 provides means for performing facialrecognition on the first image data in response to obtaining the firstimage data from the camera (e.g., front facing camera) and detection ofa face within the first image data. Priority subject selector 146determines whether a recognition criterion is met between theimage-indicative data (e.g., image data) of a detected person in the FOVand previously stored priority subject image-indicative data (e.g.,image data) of a predetermined priority subject. For example, therecognition criterion may specify a threshold probability that theperson in the FOV is a priority subject for whom corresponding prioritysubject image-indicative data (e.g., image data) are stored in thepriority subject data 165. Priority subject selector 146 can identifyone or more priority subjects. For example, priority subject selector146 can identify one of at least two detected persons as a prioritysubject. As another example, priority subject selector 146 can identifytwo of at least two detected subjects as priority subjects. Moregenerally, priority subject selector 146 can select any subset of aplurality of detected persons or animals as priority subjects.

Priority subject selector 146 initiates a facial recognition process todetermine whether any face detected by the subject detector 144 is apriority subject. In some embodiments, priority subject selector 146 caninclude initiation instructions that, when executed by processor 160,cause processor 160 to locally determine (e.g., identify) whether any ofthe subjects detected is a priority subject for whom priority subjectimage-indicative data (e.g., images) have been previously stored. Inother embodiments, priority subject selector 146 can include initiationinstructions that, when executed by processor 160, cause processor 160to transmit a request to the remote cloud processor 148. The requestcauses the cloud processor 148 to determine whether any of the detectedsubjects is a priority subject for whom priority subjectimage-indicative data have been previously stored and to respond toprocessor 160 with identification(s) of any recognized priority subject.

For example, processor 160 can determine if any detected subject in theregion of interest is a previously-identified priority subjectidentified in priority subject data 165 stored in the image capturedevice 100. Processor 160 can use priority subject image-indicative data(e.g., image data) corresponding to predetermined priority subjects tomake the determination. For example, processor 160 can compare imagedata of the subjects detected to image data of a predetermined prioritysubject using, for example, facial recognition techniques. For example,processor 160 can compare image data of the detected subjects (or dataextracted or derived therefrom) to image data of primary user (e.g.,owner) of the image capture device 100.

ROI adjuster instructions 147 are executed in processor 160 to provide ameans for selecting a region of interest of the image data correspondingto the one or more subjects determined to be priority subjects.Processor 160 can determine a region of interest of the field of view ofthe lens of camera optics and sensor 115 that includes any areas of theimage determined to include detected subjects. ROI adjuster instructions147 can include instructions that, when executed by processor 160, causeprocessor 160 to adjust a region of interest (e.g., such as onedetermined by processor 160 executing subject detector instructions 144)to include only one or more detected subjects determined to be prioritysubjects from the priority subject data 165. For example, ROI adjusterinstructions 147 can adjust a region of interest (for performing AF, AEand/or AWB) to include only detected subjects that were also identifiedpriority subjects in the priority subject data 165. In other words, theregion of interest (for performing AF, AE and/or AWB) can exclude allsubjects other than priority subjects. This allows the prioritizedpersons/subjects to have the best focus, exposure and white balance,even if there are other subjects in the FOV closer to the image capturedevice than the prioritized persons/subjects.

For example, in an image containing a large group of people in multiplerows, with a priority subject near an end of the back row, the imagecapture device will autofocus on the priority subject. In anotherexample, in a dark room (or a brightly lit space) containing a prioritysubject, the image capture device will set the exposure speed so thepriority subject has mid-tone exposure. In another example, the imagecapture device identifies the neutral tones (the whites, grays, andblacks) in the ROI containing a priority subject and then calibrates therest of the image to the temperature of the neutral colors in the ROIcontaining the priority subject.

If the FOV contains one or more priority subjects, the priority subjectdetector 146 adjusts the ROI for AF, AE, and/or AWB to include thepriority subject(s) and exclude other detected faces from considerationduring AF, AE and/or AWB. (The adjustment of the ROI for AF, AE, and/orAWB can affect the lens position, exposure speed and/or white balance,but does not exclude any area within the FOV from the image captured.)

Processor 160 can use the adjusted region of interest to perform one ormore of AF, AWB, or AE. For example, ROI adjuster instructions 147 cancause processor 160 to use the adjusted region of interest as the regionof interest when executing AF instructions 140. Similarly, ROI adjusterinstructions 147 can cause processor 160 to use the adjusted region ofinterest as the region of interest when executing AWB instructions 141or AE instructions 142.

In some examples, subject detector instructions 144, priority subjectselector instructions 146, and ROI adjuster instructions 147 areincluded in a subject based ROI engine 180. Subject based ROI engine 180can be executed by, for example, image signal processor 120, or adedicated digital signal processor (DSP) (not shown). Subject based ROIengine 180, when executed by one or more processors, can cause the oneor more processors to identify an ROI such that the one or moreprocessors, when executing 3A engine 136, perform at least one ofautomatic focus, automatic exposure, or automatic white balance of imagecapture device 100 based on the adjusted ROI.

In some examples, subject based ROI engine 180 is activated based onactivation of a camera or image capture device 100. For example,processor 160 can detect when a user activates camera optics and sensor115. Upon detecting activation, processor 160 can execute subject basedROI engine 180. In some examples, detecting activation of a camera orimage capture device 100 includes detecting activation of a front cameraof the image capture device 100 located on the same side of the deviceas the display (e.g., a front camera of a mobile phone). Activation ofthe front camera is an indication that the user intends to take aselfie, and can be used as a criterion for activating the subject basedROI engine 180.

In some examples, subject based ROI engine 180 is activated based onfingerprint detection of the primary user of image capture device 100.For example, fingerprint detection device 175 can indicate to processor160 when a user has successfully authenticated themselves via afingerprint authentication. Upon detecting successful authentication,processor 160 can execute subject based ROI engine 180. Other forms ofbiometric authentication are also contemplated, such as retina scanningor voice recognition systems and methods, and other systems and methodsas well. For example, subject based ROI engine 180 can be automaticallyactivated when the camera is active and the user performs a fingerprintauthentication or speaks a predetermined “priority subject” command foractivating subject based ROI engine 180.

Image processing instructions 143 can include instructions that causeprocessor 160 to execute one or more image processing functions such asdemosaicing, noise reduction, cross-talk reduction, color processing,gamma adjustment, image filtering (e.g., spatial image filtering), lensartifact or defect correction, image sharpening, or other imageprocessing functions.

Operating system instructions 145 can include instructions that, whenexecuted by processor 160, cause processor 160 to implement an operatingsystem. The operating system can act as an intermediary betweenprograms, such as user applications, and the processor 160. Operatingsystem instructions 145 can include device drivers to manage hardwareresources such as the camera optics and sensor 115, display 125, GPSunit 116, or transceiver 117. Instructions contained in image processinginstructions 143 discussed above can interact with hardware resourcesindirectly, through standard subroutines or application programinterfaces (APIs), that can be included in operating system instructions145. Operating system instructions 145 can then interact directly withthese hardware components. Operating system block 145 can furtherconfigure the image signal processor 120 to share information withdevice processor 150.

FIG. 2 is a flow chart of an exemplary method 200 for selecting a regionof interest in a field of view of an image capture device, such as theimage capture device 100 of FIG. 1. The method can be carried out by,for example, the ROI engine 180 of FIG. 1.

At block 202, one or more subjects among the image sensor data 201 in afield of view of an image capture device are detected to provide aregion of interest (ROI). In some embodiments, block 202 determineswhether the FOV contains a face. Block 202 can be a face detectionmodule.

At block 204, the detected face(s) (or data extracted or derivedtherefrom) is (are) compared to the priority subject image-indicativedata (e.g., images of faces) in the priority subject data 165, and adetermination is made as to whether any of the regions of interestincludes any priority subject(s). For example, facial recognition may beapplied by comparing extracted features, fiducial measurements, or thelike. If the region of interest does not include any priority subjects,execution proceeds to block 205. If the region of interest includes anypriority subjects, the method proceeds to block 206.

At block 205, the regions of interest can be set to include all facesdetected by the face detection block 202. This causes the AF to selectthe optimum lens position for all of the detected faces. For example,the total focus error for all the faces in the image is minimized (butthe individual focus error for each individual image is not necessarilyminimized).

At block 206, the region of interest is adjusted based on the prioritysubjects. For example, the region of interest for AF, AE, and/or AWB isadjusted to include subjects determined to be prioritized persons and toexclude other faces within the FOV. This will cause the camera to selecta lens position, exposure speed, and/or color temperature optimized forthe prioritized faces. The adjusted region of interest is provided asthe output region of interest.

FIGS. 3A, 3B, and 3C illustrate an exemplary image preview 300 within afield of view of the exemplary image capturing device 100 of FIG. 1. TheFOV may contain a single subject, or two or more subjects. The imagepreview 300 includes faces of a first person 302, second person 304, andthird person 306. Image capture device 100 can perform facial detectionand recognition on image data associated with image 300 to identifydetected subjects.

For example, FIG. 3B illustrates a region of interest 308 that includesimage data for all identified subjects (faces) including faces of firstperson 302, second person 304, and third person 306. If used for AF, AE,and AWB, the ROI 308 as shown in FIG. 3B is optimized for all threepersons 302, 304 and 306. The lens position is selected to minimize thetotal focus error among all three faces, but does not guarantee the bestpossible focus for any individual one of the three faces. Similarly, theexposure is selected to minimize the total exposure error among allthree faces, but does not guarantee the best possible exposure for anyindividual one of the three faces. And the white balance is selected tominimize the total color temperature error among all three faces, butdoes not guarantee the best possible color temperature for anyindividual one of the three faces.

For the example of FIG. 3B, first person 302 is a predeterminedprioritized person, for whom the priority subject data 165 (prioritizedperson list) includes one or more priority subject image-indicative data(e.g., images). The remaining subjects 304 and 306 are not prioritizedpersons. For example, image capturing device 100 can compare features ofthe face of person 302 to images stored in the prioritized person listand identify image data corresponding to first person 302 as apredetermined prioritized person. Thus, an ROI 308 as shown in FIG. 3Bcan result in an image quality for the prioritized person 302 that isnot the highest possible image quality that the image capture device 100is capable of capturing.

FIG. 3C identifies an adjusted region of interest 310 of the image 300that includes predetermined prioritized person from the priority subjectdata 165. For example, adjusted region of interest 310 includes imagedata corresponding to first person 302, but not second person 304 northird person 306. Image capture device 100 can adjust at least one ofautomatic focus, automatic exposure, or automatic white balance based onthe adjusted region of interest 310. Using the ROI 310, the imagecapture device 100 can capture an image with the best image quality forthe face of the first person 302, and permit a possible slight reductionin image quality of the other subjects 304, 306.

FIGS. 4A, 4B, and 4C illustrate an exemplary image preview 400 in afield of view of the exemplary image capture device 100 of FIG. 1. Theimage 400 includes images of a first person 402, second person 404, andthird person 406. Image capture device 100 can perform facial detectionand recognition on image data associated with image 400 to identifydetected subjects.

For example, FIG. 4B illustrates a region of interest 408 that includesimage data for all identified subjects including first person 402,second person 404, and third person 406. For this example, assume thatfirst person 402 and third person 406 are each predetermined prioritizedpersons, but the second person 404 is not included in the prioritysubject data 165. For example, image capture device 100 can includeimage data or other priority subject image-indicative data correspondingto first person 402 and third person 406 in the priority subject data165. If used for AF, AE, and AWB, the ROI 408 as shown in FIG. 4B isoptimized for all three persons 402, 404 and 406. The lens position isselected to minimize the total focus error among all three faces 402,404 and 406, but does not guarantee the best possible focus, exposure orwhite balance for persons 402 and 406. Thus, the ROI 408 could increasethe image quality of the subject 404 at the expense of the prioritysubjects 402 and 406.

FIG. 4C identifies an adjusted region of interest 410 of the image 400that includes predetermined prioritized persons 402 and 406. Forexample, adjusted region of interest 410 includes image datacorresponding to the faces of first person 402 and third person 406, butnot second person 404. Image capture device 100 can adjust at least oneof automatic focus, automatic exposure, or automatic white balance basedon the adjusted region of interest 410. Adjusted region of interest 410can provide the optimum focus, exposure and white balance with minimumtotal error for the faces of the prioritized persons 402, 406

In some embodiments, the processor 160 can train a neural network withseveral images of the primary user, and use the neural network todetermine whether the face in the FOV of the image capture device 100belongs to the primary user. For example, the processor 160 can useunsupervised learning to train a variational autoencoder with images ofthe priority subject, and use outlier detection to determine if asubject in the FOV of the camera belongs to the class of the trainingsubject (corresponding to the priority subject). The outlier detectioncan use a random sampling consensus (RANSAC) or random forest technique.

In some examples, processor 160 compares image data (or data extractedor derived therefrom) for each detected subject in the FOV to previouslystored priority subject image-indicative data (e.g., image data) ofpriority subjects (e.g., persons or animals) in a priority subject data165. (As discussed herein, the priority subject data 165 identifies oneor more subjects for whom the storage medium 110 stores priority subjectimage-indicative data (e.g., one or more images or images plus text),and who are to be given priority when selecting the ROI for an AF, AE orAWB operation.) The priority subject data 165 can include prioritysubject image-indicative data (e.g., image data) for one or morepredetermined prioritized persons, and can be stored as priority subjectdata 165 in the non-transitory, machine-readable storage medium 110, forexample. The priority subject data 165 can include a plurality oftraining images (or training data corresponding to other prioritysubject image-indicative data) for each predetermined priority subject.The priority subject data 165 may also include text, such as a nameand/or bibliographic data associated with one or more of the prioritizedsubjects.

In some embodiments, the priority subject data 165 has a single prioritylevel; faces in the priority subject data 165 are given priority overfaces that are not included in the priority subject data 165. In otherembodiments, the priority subject data 165 has multiple priority levels.If the FOV contains persons included in priority subject data 165 withtwo or more different priority levels and/or persons who are notincluded in the priority subject data 165, then the ROI for AF, AE andAWB can be selected to only include the face of the person with thehighest priority level. The persons with lower priority levels andpersons not included in the priority subject data 165 can be excludedfrom the ROI.

FIG. 5 is a flow chart of an exemplary method 500 that can be carriedout by, for example, image capture device 100. In the example of FIG. 5,the image capture device 100 is configured to detect and recognize whenthe camera's primary user (e.g., owner) is within the FOV, and to use anROI narrowly enclosing the face of the camera primary user for AF, AEand AWB, regardless of whether the FOV contains additional people. Thiswill cause the primary user to have the best image quality possible,even if there are other faces closer to the camera than the primaryuser.

At block 502, an priority subject image-indicative data (e.g., an image)of the primary user's face is stored in a non-transitory,machine-readable storage medium 110 (FIG. 1) in the image capture device100. For example, the image can be captured via a selfie and stored instorage medium 110. Alternatively, the image can be received from anexternal camera (not shown) and saved in the photographs folder (notshown) of the image capture device 100. In some embodiments, the imagecapture device 100 has a folder containing priority subject data 165(FIG. 1) for storing priority training images of the primary user.

At block 504, an image capture device 100, detects activation of a frontcamera on a front face of the image capture device, where the front facehas a display. For example, if the image capture device 100 is a mobilephone, the front camera is the camera on the face of the phone havingthe display. In response to activation of the camera located on thefront face of the image capture device, block 504 initiates a “selfiemode”, in which the image capture device 100 searches for prioritysubjects in the FOV of the camera.

At block 506, the subject detector 144 (FIG. 1) applies face detectionand facial recognition to search for at least one face within the firstimage data within the FOV (i.e., preview image data). In someembodiments, while operating in the selfie mode, the priority subjectselector 146 attempts to match the detected face in the first image datawithin the FOV of the image capture device against the face of theprimary user. In other embodiments, the priority subject selector 146attempts to match the detected face in the first image data within theFOV of the image capture device against faces of all priority users.

At block 508, a determination is made as to whether the face of theprimary user was recognized. For example, the priority subject selector146 can compare a face within the first image data within the FOVagainst one or more stored images of the primary user's face stored inthe storage medium 110. If the primary user is recognized, executionproceeds to block 512. If the primary user is detected, the methodproceeds to block 510.

At block 510, the ROI used for at least one of automatic focus,automatic exposure, or automatic white balance is adjusted based onidentification of one or more faces within the field of view of thedevice that includes the primary user's face. For example, one or moreof automatic focus, automatic exposure, or automatic white balanceparameters can be adjusted. Then AF, AE and/or AWB is performed usingthe adjusted parameters.

At block 512, one or more preview settings are adjusted in accordancewith the new parameters computed by AF, AE and/or AWB based on theadjusted ROI.

At block 514, the image capture device 100 captures a second image datawithin the FOV of the image capture device 100, based on the automaticfocus, automatic exposure, or automatic white balance using the selectedregion of interest. For example, if the primary user is recognized bythe priority subject selector 146 (FIG. 1), the image of the primaryuser within the FOV is captured using the one or more adjusted automaticfocus, automatic exposure, or automatic white balance.

FIG. 6 is a flowchart 600 of another exemplary method that can becarried out by, for example, the image capture device 100 of FIG. 1. Inthe example of FIG. 6, the image capture device 100 has a prioritysubject data 165 containing a set of images or other priority subjectimage-indicative data showing any desired number of prioritized persons(e.g., members of the primary user's family, the family pet). In someembodiments, the priority subject data 165 contains a plurality ofimages or other priority subject image-indicative data for each prioritysubject. In other embodiments, the priority subject data 165 alsoincludes bibliographic information associated with each of the prioritysubjects. The ROI adjuster 147 (FIG. 1) adjusts the ROI to include eachperson in the priority subject data 165 within the FOV, and to excludeany persons who are not in the priority subject data 165.

At block 603, a region of interest is determined based on all subjectsin a field of view of a camera. For example, if subject detector 144(FIG. 1) determines that there are a plurality of faces (e.g., faces402, 404, 406, FIG. 4) in the FOV of the image capture device 100, theROI is set to ROI 408, so as to include all of the identified faces inthe FOV.

At block 604, subject detector 144 (FIG. 1) determines whether anysubjects in the field of view of the camera have priority. For example,subject detector 144 compares image data of the subjects in the regionof interest (or data extracted or derived therefrom) to priority subjectimage-indicative data (e.g., image data) of predetermined prioritizedpersons in a priority subject data 165. If no subjects have priority,the method proceeds to block 608. Otherwise, if any subjects havepriority, the method proceeds to block 606.

At block 606, ROI adjuster 147 (FIG. 1) adjusts the region of interestbased on the determination that one or more of the subjects in the FOVhave priority. For example, the region of interest can be adjusted toinclude subjects having priority, but exclude subjects not havingpriority. The method then proceeds to block 608.

At block 608, the 3A engine 136 performs at least one of AF, AE, or AWBbased on the adjusted region of interest. If any subjects weredetermined to have priority in block 604, then the region of adjustedinterest (e.g., 410, FIG. 4C) includes the regions of interest 410 (FIG.4C) determined in block 606. Otherwise, the region of interest 408 (FIG.4B) includes all the regions of interests 404, 408 (FIG. 4B) determinedin block 603.

At block 610, the camera optics and sensor 115 (FIG. 1) capture animage. For example, the image can be captured using the AF, AE, and AWBoptimized for the adjusted ROI 410 (FIG. 4C).

In other embodiments, the priority subject data 165 can include two ormore priority levels. The priority subject data 165 can assign one ormore predetermined prioritized persons to respective priority levels.For example, a first predetermined prioritized person (e.g., the primaryuser's child) can be assigned a highest priority, a second predeterminedprioritized person (e.g., the primary user) can be associated with anext highest priority, and so on. If the FOV of the image capture device100 includes a higher priority subject and a lower priority subject, theROI adjuster 147 includes the higher priority subject in the ROI (forAF, AE and/or AWB) and excludes the lower priority subject. If the FOVof the image capture device 100 includes a lower priority subject and anunknown subject, the ROI adjuster 147 includes the lower prioritysubject in the ROI and excludes the unknown subject. In general, if theFOV contains subjects with two or more priority levels, the ROI adjuster147 includes the subject having the highest priority level in the ROIand excludes any other persons within the FOV from the ROI.

In some examples, some prioritized persons can be predetermined orselected in advance by a user. Other prioritized persons are notpre-selected, but may be automatically added to the priority subjectdata 165 in response to determining that the priority subject selector146 determining that the same subject has been captured at least athreshold number of times within a predetermined time interval.

In some examples, processor 160 can determine whether any of thedetected subjects in the image have different priority levels amongpersons of the priority subject data 165. For example, processor 160 canidentify two priority subjects in the image, where one subject isassociated with the highest priority level in the priority subject data165, and another subject is associated with the next highest prioritylevel in the priority subject data 165. In some examples, processor 160determines if a configurable number of detected subjects are prioritizedpersons. For example, the configurable number of detected subjects canbe set by a user of image capture device 100 via input device 170.

In some examples, the priority subject data 165 includes predeterminedprioritized persons associated with a priority level, and dynamicallyselected prioritized persons automatically added by virtue of beingcaptured a threshold number of times within a predetermined period. Inother embodiments, the priority subject data 165 contains two separatelists or folders: static priority subject data 165 containing thepriority subjects previously selected (by the primary user) and atemporary list or folder containing automatically added prioritysubjects. Processor 160 can determine whether any of the detectedsubjects is a previously selected priority subject with a previouslyselected priority level. If no detected subjects within the FOV areidentified as being prioritized persons, processor 160 can dynamicallydetermine whether any detected subjects have been captured a thresholdnumber of times within a predetermined period. Dynamically selectedsubjects can be added to a list or folder of temporary priority subjectdata 165 for a predetermined period of time.

In some examples, processor 160 can determine if up to a configurablenumber of subjects detected are priority subjects. A user of imagecapture device 100 can set the configurable number of priority subjectsvia input device 170.

In some examples, priority subject selector instructions 146 can includeinstructions that, when executed by processor 160, cause processor 160to identify (e.g., select) at least two detected subjects aspredetermined prioritized persons. For example, if two out of three ormore persons within the FOV are priority subjects in the prioritysubject data 165, the ROI for performing AF, AE and AWB includes thefaces of the two priority subjects, and any other persons within the FOVare excluded from the ROI for AF, AE, and AWB.

In some examples, priority subject selector 146 can include instructionsthat, when executed by processor 160, cause processor 160 to add adynamically selected priority subject to the temporary priority subjectdata, based on a location of the image capture device 100 at the time ofimage capture. This capability can allow the user to select a location(e.g., the user's home or a vacation spot) at which every personcaptured is added to the temporary priority subject data 165. Theselected location can be, for example, the person's home address, homestate, favorite place, or any other location. For example, processor 160can obtain a location of image capture device 100 by receiving locationdata from GPS unit 116, and compare the current location to the selectedlocation. Location data can include, for example, latitude and longitudeinformation. In other embodiments, the processor 160 can determine thecurrent location of the image capture device 100 based on the locationof a WiFi access point, triangulation from beacons, signal from a “smartwatch”, or the like. If processor 160 determines that the location ofimage capture device 100 is at or near the selected location, thepriority subject selector 146 can add the person to the priority subjectdata 165. For example, any person within the FOV of the image capturedevice 100 while the image capture device is located at the user's homewill be added to the temporary priority subject data 165.

In some embodiments, the image capture device maintains separatepriority subject data 165 for persons associated with each of aplurality of selected locations. In some examples, a person can beassociated with more than one location. In some examples, locationinformation associated with priority subjects can be included in thepriority subject data 165 stored in storage medium 110. For example, auser can provide location information to be associated with prioritysubjects to image capture device 100 via input device 170.

In some examples, priority subject selector instructions 146 can includeinstructions that, when executed by processor 160, cause processor 160to recognize at least one priority subject based on image datacorresponding to the priority subject obtained over a network. Forexample, transceiver 117 is operable to transmit data to, and receivedata from, a network. The network can be a WiFi® network, a cellularnetwork such as a 3GPP® network, a Bluetooth® network, or any othersuitable network. The network can provide access to, for example, theInternet. Processor 160 is in communication with transceiver 117, andcan request and receive image data corresponding to one or morepredetermined prioritized persons from the network. For example,processor 160 can search social media, a search engine, or an onlineencyclopedia over the network for images of a nearby person based on thelocation of the image capture device, obtain one or more images of thenearby person from social media, search engine, or online encyclopediavia the network, and add the nearby person to the priority subject data165. Processor 160 can add the one or more obtained images (or otherimage-indicative data based on the obtained images) of the nearby personto a folder storing images of persons identified in the priority subjectdata 165, for example.

In other embodiments, the image capture device 100 allows the user toadd the name of an individual to the priority subject data 165, and theimage capture device automatically retrieves training images associatedwith the named individual from a search engine, social media and/or theuser's photo library in the image capture device. The image capturedevice 100 uses machine learning to learn the person's face, andrecognizes the person's face as being associated with a prioritizedperson when the face is within the FOV of the image capture device 100.

In some examples, priority subject selector instructions 146 can includeinstructions that, when executed by processor 160, cause processor 160to add at least one predetermined prioritized person to the prioritysubject data 165, based on how often the predetermined prioritizedperson has been detected as a subject in the FOV of the image capturedevice 100. For example, processor 160, when executing subject detectorinstructions 144, can associate a time of capture with each detectedsubject. Processor 160 can store image data associated with eachdetected subject, along with a time of capture, as part of the prioritysubject data 165 stored in storage medium 110. If the same subject isdetected, processor 160 can update the most recent time of captureassociated with that subject. Processor 160 can also maintain a count ofhow many times the subject has been detected. For example, the count canbe based on how often the subject was detected over a period of time(e.g., over the last month, week, day, etc.). In some examples, thecount can be configured by a user via input device 170. The count canalso be stored as part of priority subject data 165 in storage medium110.

Processor 160, when executing priority subject selector instructions146, can read priority subject data 165 from storage 110 to determinehow often the subject in the FOV has been detected over a recent periodof time. If the subject has been detected more than a threshold numberof times during that period, processor 160 can add the person to thepriority subject data 165. If the image capture device 100 does notassociate the person with a name, the image capture device can generatea temporary name for the subject. If the subject has not been detectedmore than the threshold number of times during the period, the subjectis not added to the priority subject data 165. For example, processor160 can determine that at least one of one or more subjects in the fieldof view of the image capture device 100 has previously been photographedusing the image capture device (and previously detected as a subject).Processor 160 can store image data (or other image-indicative data basedon the image data) corresponding to the previously captured subject tothe priority subject data 165 in storage medium 110.

Processor 160 can be configured to add images or other image-indicativedata of a person to the priority subject data 165 (or a temporarypriority subject data 165) if a threshold number of images of the sameperson are determined to be captured within a predetermined period oftime. For example, processor 160 can add images or otherimage-indicative data of a person to the priority subject data 165 (or atemporary priority subject data 165) if the same person is captured inten images within one minute. In some embodiments, the temporarypriority subject data 165 can be purged upon passage of a predeterminedperiod (e.g., one week) after being added to the temporary prioritysubject data 165. In all other respects the temporary priority subjectdata 165 can be structured and used the same way as the static prioritysubject data 165 described herein.

FIG. 7 is a flowchart 700 of another exemplary method that can becarried out by, for example, the image capture device 100 of FIG. 1. Inthe example of FIG. 7, instead of using images captured by the user tospecify predetermined prioritized persons, the priority subject selector146 (FIG. 1) allows the user to input the name of a subject (e.g., awell-known public figure) whom the user wishes to photograph. Thepriority subject selector 146 can obtain images of the predeterminedperson from the Internet using a search engine (e.g., Google). The imagecapture device can save the images or other priority subjectimage-indicative data into a folder within priority subject data 165 for“temporary prioritized persons”. The saved images can be used foradjusting the ROI for AF, AE and AWB, in the manner described above forthe primary user of the image capture device 100. When the image capturedevice is used, the subject in the FOV can be compared to the temporaryprioritized person; if the FOV contains the temporary prioritizedperson, the priority subject selector 146 selects an ROI containing theface of the temporary predetermined person.

At block 701, priority subject selector 146 retrieves the name of apriority subject from the priority subject data 165. In someembodiments, priority subject selector 146 determines that the number ofimages or views (or other priority subject image-indicative data) of thesubject is insufficient for accurate recognition. In other embodiments,the priority subject data 165 includes a separate folder or storage areafor identifying names of priority subjects without accompanying imagesor other priority subject image-indicative data.

At block 702, subject detector 144 tries again to obtain images of thenamed priority subject using a search engine (e.g., Google or Yahoo!) orusing social media (e.g., Facebook).

At block 703, subject detector 144 performs facial detection on imagedata within a field of view of an image capture device 100 to detect oneor more subjects.

At block 704, priority subject selector 146 determines whether any ofthe subjects in the field of view of the image capture device 100 is apredetermined person.

At block 706, a region of interest of image data is identified thatcontains one or more of the subjects determined to be a predeterminedperson, and excludes one or more subjects determined not to bepre-determined persons.

At block 708, at least one of AF, AE, or AWB of the image capture deviceis performed based on the identified region of interest. For example,the AF, AE, and AWB can all be performed based on the identified regionof interest.

FIG. 8 is a flow chart of an exemplary method 800 that can be carriedout by image capture device 100. In the example of FIG. 8, the prioritysubject detection mode is activated when the primary user's fingerprintis detected while the front camera of the image capture device 100 isactivated.

At block 802, an image of the device primary user's face (or otherpriority subject image-indicative data derived therefrom) is stored in anon-transitory, machine-readable storage medium 110 (FIG. 1) in theimage capture device 100. For example, the image can be captured via aselfie and stored in storage medium 110 or, the image can be receivedfrom an external camera (not shown) and saved in the priority subjectdata 165 of image capture device 100.

At block 804, a front camera of image capture device 100 is activated,where the front face is the face having a display.

At block 806, biometric information (fingerprint, iris, or voice) of theuser is detected. For example, if the image capture device 100 is a cellphone, a fingerprint detection device 175 can be located on the homebutton of the phone. In response to the fingerprint detection device 175(FIG. 1) of image capture device 100 detecting the primary user'sfingerprint while the front camera is active, execution automaticallypasses to block 808, activating facial recognition. Similarly, in otherembodiments, execution passes to block 808 in response to the phonedetecting the primary user's iris or voice.

At block 808, a determination is made as to whether a face of theprimary user is in the FOV. For example, the priority subject selector146 can compare a face within the FOV against one or more stored imagesof the primary user's face stored in the storage medium 110. If theprimary user is detected, execution proceeds to block 812. If theprimary user is detected, the method proceeds to block 810.

At block 810, the ROI used for at least one of automatic focus,automatic exposure, or automatic white balance is adjusted based onidentification of the primary user's face.

At block 812, one or more preview settings are adjusted in accordancewith the new parameters computed by AF, AE and/or AWB based on theadjusted ROI.

At block 814, an image is captured. For example, if the primary user isrecognized by the priority subject selector 146 (FIG. 1), the image iscaptured using the one or more adjusted automatic focus, automaticexposure, or automatic white balance.

FIG. 9 shows additional blocks that can be added to the method of FIG.6, in some embodiments. The additional blocks in FIG. 9 designate aperson in a field of view of the image capture device as a prioritysubject in response to determining that the image capture device is in apredetermined location. Blocks 601 and 602 of FIG. 9 are executed priorto execution of block 603 (FIG. 6).

At block 601, a determination is made whether the image capture device100 is located in a predetermined location entered into the prioritysubject data 165. For example, the predetermined location can beidentified by name, address or by GPS coordinates. If the image capturedevice 100 is located in a predetermined location, block 602 isexecuted. If the image capture device 100 is not located in apredetermined location, execution passes to step 603 of FIG. 6.

At block 602, image data (or other image-indicative data extracted orderived therefrom) for each face in the FOV are added to the prioritysubject data 165 in the image capture device 100. After block 602,execution passes to block 603, FIG. 6.

Using the method of FIGS. 9 and 6, the user can instruct the imagecapture device to make every face detected in a predetermined location(e.g., the user's home, a vacation spot, a place of employment, etc.) apriority subject. Subsequently, when these priority subjects are withinthe FOV of the image capture device, they will be included in the ROIfor AE, AF, and/or AWB.

Although the methods described above are with reference to theillustrated flowcharts, many other ways of performing the actsassociated with the methods can be used. For example, the order of someoperations may be changed, and some embodiments can omit one or more ofthe operations described and/or include additional operations.

In addition, the methods and system described herein can be at leastpartially embodied in the form of computer-implemented processes andapparatus for practicing those processes. The disclosed methods may alsobe at least partially embodied in the form of tangible, non-transitorymachine-readable storage media encoded with computer program code. Forexample, the methods can be embodied in hardware, in executableinstructions executed by a processor (e.g., software), or a combinationof the two. The media may include, for example, RAMs, ROMs, CD-ROMs,DVD-ROMs, BD-ROMs, hard disk drives, flash memories, or any othernon-transitory machine-readable storage medium. When the computerprogram code is loaded into and executed by a computer, the computerbecomes an apparatus for practicing the method. The methods may also beat least partially embodied in the form of a computer into whichcomputer program code is loaded or executed, such that, the computerbecomes a special purpose computer for practicing the methods. Whenimplemented on a general-purpose processor, the computer program codesegments configure the processor to create specific logic circuits. Themethods may alternatively be at least partially embodied in applicationspecific integrated circuits for performing the methods.

Although the subject matter has been described in terms of exemplaryembodiments, it is not limited thereto. For example, the claims are notlimited to the exemplary embodiments. For example, some claimedimplementations can have different features than that in the exemplaryembodiments. In addition, changes can be made without departing from thespirit of the disclosure. For example, features of the exemplaryembodiments can be incorporated in different systems (e.g., devices) andmethods. Thus, the appended claims should be construed broadly, toinclude other variants and embodiments, which can be made by thoseskilled in the art having the benefit of the present disclosures.

We claim:
 1. A method for controlling an image capture device,comprising: obtaining first image data from a camera of the imagecapture device, the first image data representing one or more subjectswithin a field of view of the image capture device; performing facialrecognition on the first image data based on priority subjectimage-indicative data comprising coefficient data for a particularsubject, wherein performing the facial recognition comprises: applying atrained neural network to the coefficient data and to the first imagedata; and identifying a face of the particular subject based on theapplication of the trained neural network to the coefficient data and tothe first image data; determining, based on the facial recognition, thatthe one or more subjects include the particular subject; selecting aregion of interest corresponding to at least the particular subject; andprocessing a second image based on the selected region of interest. 2.The method of claim 1, wherein performing the facial recognitioncomprises detecting a face within the first image data.
 3. The method ofclaim 1, wherein: the priority subject image-indicative data comprises aplurality of stored facial features for the particular subject; andperforming the facial recognition comprises: extracting facial featuresfrom the first image data; and comparing the stored plurality of storedfacial features with the extracted facial features.
 4. The method ofclaim 1, wherein: the priority subject image-indicative data comprises aplurality of stored intensity values for the particular subject; andperforming the facial recognition comprises: determining intensityvalues from the first image data; and comparing the plurality of storedintensity values with the determined intensity values.
 5. The method ofclaim 4, wherein the neural network is trained to identify members of apredetermined set of priority subjects comprising the particularsubject.
 6. The method of claim 1, wherein processing the second imagedata includes capturing the second image data.
 7. The method of claim 6,wherein the second image data is captured using at least one of anautomatic focus, automatic exposure, or automatic white balance settingdetermined based, at least in part, on the selected region of interest.8. The method of claim 1, comprising: receiving a user selection; andobtaining the priority subject image-indicative data based on the userselection.
 9. The method of claim 1, wherein the region of interestincludes each of the one or more subjects.
 10. The method of claim 1,wherein selecting the region of interest comprises adjusting the regionof interest to include the particular subject.
 11. The method of claim10, wherein the selecting further comprises adjusting the region ofinterest to exclude at least one of the one or more subjects.
 12. Animage capture device comprising: a memory containing priority subjectimage-indicative data corresponding to at least one particular subject;and at least one processor coupled to the memory for accessing thepriority subject image-indicative data, the at least one processor beingconfigured to: obtain first image data from a camera of the imagecapture device, the first image data representing one or more subjectswithin a field of view of the image capture device; perform facialrecognition on the first image data based on priority subjectimage-indicative data comprising coefficient data for a particularsubject, wherein the at least one processor is further configured to:apply a trained neural network to the coefficient data and to the firstimage data; and identify a face of the particular subject based on theapplication of the trained network to the coefficient data and to thefirst image data; determine, based on the facial recognition, that theone or more subjects include the particular subject; select a region ofinterest corresponding to at least the particular subject; and process asecond image based on the selected region of interest.
 13. The imagecapture device of claim 12, wherein the at least one processor isfurther configured to detect a face within the first image data.
 14. Theimage capture device of claim 12, wherein: the priority subjectimage-indicative data comprises a plurality of stored facial featuresfor the particular subject; and the at least one processor is furtherconfigured to: extract facial features from the first image data; andcompare the stored plurality of stored facial features with theextracted facial features.
 15. The image capture device of claim 12,wherein: the priority subject image-indicative data comprises aplurality of stored intensity values for the particular subject; and theat least one processor is further configured to: determine intensityvalues from the first image data; and compare the plurality of storedintensity values with the determined intensity values.
 16. The imagecapture device of claim 12, wherein the neural network is trained toidentify members of a predetermined set of priority subjects comprisingthe particular subject.
 17. The image capture device of claim 12,wherein the at least one processor is further configured to capture thesecond image data.
 18. The image capture device of claim 17, wherein theat least one processor is further configured to capture the second imagedata using at least one of an automatic focus, automatic exposure, orautomatic white balance setting determined based, at least in part, onthe selected region of interest.
 19. The image capture device of claim12, wherein the at least one processor is further configured to: receivea user selection; and obtain the priority subject image-indicative databased on the user selection.
 20. The image capture device of claim 12,wherein the region of interest includes each of the one or moresubjects.
 21. The image capture device of claim 12, wherein the at leastone processor is further configured to adjust the region of interest toinclude the particular subject.
 22. The image capture of claim 21,wherein the at least one processor is further configured to adjust theregion of interest to exclude at least one of the one or more subjects.23. A non-transitory computer-readable storage medium comprisingcomputer-executable instructions stored tangibly thereon, such that whenthe instructions are executed by one or more processors, theinstructions cause the one or more processor to: obtain first image datafrom a camera of the image capture device, the first image datarepresenting one or more subjects within a field of view of the imagecapture device; perform facial recognition on the first image data basedon priority subject image-indicative data comprising coefficient datafor a particular subject, wherein performing the facial recognitioncomprises: applying a trained neural network to the coefficient data andto the first image data; and identifying a face of the particularsubject based on the application of the trained network to thecoefficient data and to the first image data; determine, based on thefacial recognition, that the one or more subjects include the particularsubject; select a region of interest corresponding to at least theparticular subject; and process a second image based on the selectedregion of interest.
 24. The non-transitory computer-readable storagemedium of claim 23 wherein processing the second image data comprisescapturing the second image data using at least one of an automaticfocus, automatic exposure, or automatic white balance setting determinedbased, at least in part, on the selected region of interest.